Method, apparatus, and storage medium for processing image in a virtual reality system

ABSTRACT

The present application discloses a method, apparatus, and storage medium for processing an image in a virtual reality system. The method comprises: splitting an image to be rendered at a first timing in the virtual reality system into a first region and a second region; rendering the first region of the image to be rendered according to first gesture data of a user at the first timing; rendering the second region of the image to be rendered according to second gesture data at a second timing at the end of the rendering of the first region.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to Chinese Patent ApplicationNo. 201910001464.7, filed on Jan. 2, 2019, which is incorporated hereinby reference and used for all purpose.

TECHNICAL FIELD

The present application relates to a method, apparatus, and storagemedium for processing an image in a virtual reality system.

BACKGROUND

With the rapid development of Virtual Reality (VR) technology, VRdevices have gradually entered the people's field of vision, and aregetting more and more applications. However, in practical applications,VR devices have a certain delay in sensing human body movements, so thatan image viewed on a VR device by a user is not synchronized with thebody movement of the user, which is liable to cause human discomfort.

Therefore, there is a need for providing a technology for improvingimage processing of VR devices.

SUMMARY

In an aspect of the present disclosure, there is provided a method forprocessing an image in a virtual reality system, comprising: splittingan image to be rendered at a first timing in the virtual reality systeminto a first region and a second region; rendering the first region ofthe image to be rendered according to first gesture data of a user atthe first timing; and rendering the second region of the image to berendered according to second gesture data at a second timing at the endof the rendering of the first region.

In an example, the first timing is a timing at which a vertical syncsignal is generated in the virtual reality system.

In an example, the method further comprises: after rendering the secondregion of the image to be rendered, performing a TimeWarp process on animage corresponding to the first region as rendered and an imagecorresponding to the second region as rendered.

In an example, rendering the first region of the image to be renderedcomprises rendering the first region with a first resolution; renderingthe second region of the image to be rendered comprises rendering thesecond region with a second resolution; and the first resolution islower than the second resolution.

In an example, splitting an image to be rendered at a first timing inthe virtual reality system into a first region and a second regioncomprises: performing subject recognition on the image to be rendered atthe first timing to determine a region in the image to be rendered atwhich a subject is located; and determining the region in the image tobe rendered at which the subject is located as the second region.

In an example, splitting an image to be rendered at a first timing inthe virtual reality system into a first region and a second regioncomprises: performing subject recognition on the image to be rendered atthe first timing to determine a region in the image to be rendered atwhich a subject is located; and determining the region in the image tobe rendered at which the subject is located as the second region.

In an example, splitting an image to be rendered at a first timing inthe virtual reality system into a first region and a second regioncomprises: determining a gaze point location of the user at the firsttiming as a reference point position of the second region; anddetermining the second region according to a preset extension mode andthe reference point position.

In an example, splitting an image to be rendered at a first timing inthe virtual reality system into a first region and a second regioncomprises: determining a gaze point location of the user at the firsttiming as a reference point position of the second region; anddetermining the second region according to a preset extension mode andthe reference point position.

In an example, splitting an image to be rendered in the virtual realitysystem into a first region and a second region comprises: determining afield of view of the user according to a distance between a current gazepoint location of the user and a current location of the user; anddetermining the second region according to the field of view of the userand the current gaze point location.

In an example, the method further comprises: before rendering the secondregion of the image to be rendered, determining that a duration ofrendering of the first region is less than a first threshold.

In an example, the method further comprising: before rendering thesecond region of the image to be rendered, determining that a distancebetween a gaze point location corresponding to the second gesture dataand a gaze point location at the first timing is less than a secondthreshold.

In another aspect of the present disclosure, there is provided anapparatus for processing an image in a virtual reality system,comprising: a central processing unit configured to split an image to berendered at a first timing in the virtual reality system into a firstregion and a second region; a graphics processing unit configured to:render the first region of the image to be rendered according to firstgesture data of a user at the first timing; and render the second regionof the image to be rendered according to second gesture data at a secondtiming at the end of the rendering of the first region.

In an example, the graphics processing unit is further configured to:perform a TimeWarp process on an image corresponding to the first regionas rendered and an image corresponding to the second region as renderedafter rendering the second region of the image to be rendered.

In an example, in order to render the first region of the image to berendered, the graphics processing unit is configured to render the firstregion with a first resolution; in order to render the second region ofthe image to be rendered, the graphics processing unit is furtherconfigured to render the second region with a second resolution; and thefirst resolution is lower than the second resolution.

In an example, in order to split an image to be rendered at a firsttiming in the virtual reality system into a first region and a secondregion, the central processing unit is further configured to: performsubject recognition on the image to be rendered at the first timing todetermine a region in the image to be rendered at which a subject islocated; and determine the region in the image to be rendered at whichthe subject is located as the second region.

In an example, in order to split an image to be rendered at a firsttiming in the virtual reality system into a first region and a secondregion, the central processing unit is further configured to: determinea gaze point location of the user at the first timing as a referencepoint position of the second region; and determine the second regionaccording to a preset extension mode and the reference point position.

In an example, in order to split an image to be rendered in the virtualreality system into a first region and a second region, the centralprocessing unit is further configured to: determine a field of view ofthe user according to a distance between a current gaze point locationof the user and a current location of the user; and determine the secondregion according to the field of view of the user and the current gazepoint location.

In an example, the central processing unit is further configured to:before rendering the second region of the image to be rendered,determine that a duration of rendering of the first region is less thana first threshold.

In an example, the central processing unit is further configured to:before rendering the second region of the image to be rendered,determine that a distance between a gaze point location corresponding tothe second gesture data and a gaze point location at the first timing isless than a second threshold.

In yet another aspect of the present disclosure, there is provided acomputer-readable storage medium on which a computer program is stored,which when executed by a processor caused the processor to: split animage to be rendered at a first timing in the virtual reality systeminto a first region and a second region; render the first region of theimage to be rendered according to first gesture data of a user at thefirst timing; and render the second region of the image to be renderedaccording to second gesture data at a second timing at the end of therendering of the first region.

Additional aspects and advantages of the present application will partlybe presented in the following description, partly become apparent in thefollowing description or be appreciated in practicing of the presentapplication.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or additional aspects and advantages of the presentdisclosure will become apparent or easy to understand from the followingdetailed description taken in conjunction with the accompanyingdrawings, wherein:

FIG. 1 is a schematic diagram of a process in which a virtual realitydevice displays a corresponding image according to a user movement inthe related art;

FIG. 2 is a schematic flowchart of a method for processing an image in avirtual reality system according to an embodiment of the presentapplication;

FIG. 3 is a schematic flowchart of splitting an image to be rendered ata first timing into a first region and a second region according to alocation of a subject in the image to be rendered according to anembodiment of the present application;

FIG. 4 is a schematic flowchart of a method for processing an image in avirtual reality system according to another embodiment of the presentapplication;

FIG. 5 is a schematic flowchart of splitting an image to be rendered ata first timing into a first region and a second region according to alocation of the gaze point of a user according to an embodiment of thepresent application;

FIG. 6 is a diagram of splitting an image to be rendered at a firsttiming into a first region and a second region according to a locationof the gaze point of a user according to an embodiment of the presentapplication;

FIG. 7 is a schematic flowchart of splitting an image to be rendered ata first timing into a first region and a second region according to alocation of the gaze point of a user according to another embodiment ofthe present application;

FIG. 8 is a schematic flowchart of a method for processing an image in avirtual reality system according to a further embodiment of the presentapplication;

FIG. 9 is a schematic diagram of a process of rendering and displayingan image to be rendered in the related art;

FIG. 10 is a schematic diagram of a process of rendering and displayingan image to be rendered according to an embodiment of the presentapplication;

FIG. 11 is a schematic structural diagram of an apparatus for processingan image in a virtual reality system according to an embodiment of thepresent application;

FIG. 12 is a schematic structural diagram of an apparatus for processingan image in a virtual reality system according to another embodiment ofthe present application; and

FIG. 13 is a schematic structural diagram of an apparatus for processingan image in a virtual reality system according to a further embodimentof the present application.

DETAILED DESCRIPTION

A detailed description of embodiments of the present application will begiven below. Illustrative embodiments are shown in the drawings, inwhich the similar reference signs are used throughout to represent thesame or similar elements or elements having the same or similarfunctions. The embodiments described with reference to the drawings areillustrative, which are merely used to interpret the presentapplication, but cannot be understood as limitation to the presentapplication.

In view of the problem of causing discomfort by out-of-synchronizationbetween an image viewed by a human on a VR device and a body movement ofthe human body in the related art, an embodiment of the presentapplication provides a method for processing an image in a VR system.

In the embodiment of the present application, an image to be rendered ata first timing in the VR system is split into a first region and asecond region; the first region of the image to be rendered is renderedaccording to first gesture data at the first timing; and the secondregion of the image to be rendered is rendered according to secondgesture data at a second timing at the end of the rendering of the firstregion to obtain a rendered image. Therefore, by using gesture data atdifferent timings, time-division rendering is performed on differentregions of the image to obtain a rendered image. Therefore, the timeinterval between a timing at which the rendered image is displayed and atiming at which the gesture data is collected for rendering is shortenedas much as possible, which effectively reduces the delay of the imageand improves the user experience.

In order to illustrate the embodiment of the present application moreclearly, description will be given below for the source of delay presentin the VR system of a VR device that senses a human movement to displaya corresponding image.

As shown in FIG. 1, when the user turns his/her head, the VR devicegenerates a corresponding digital signal according to first gesture ofthe head rotation of the user, and transmits the corresponding digitalsignal to a processing terminal (for example, a computer terminal).Thereafter, the processing terminal processes the signal to generate animage corresponding to the head gesture, and then displays the processedimage to the user until the display is completed, however at this timethe gesture of the user's head may not be identical with the firstgesture. Since the processing of each of the above steps requires acertain amount of time, and the user's head may be in continuousrotation, the displayed image viewed at a certain gesture to which thehead rotates is actually an image corresponding to a gesture at aprevious timing.

In order to reduce the delay in a VR device, an embodiment of thepresent application reduces out-of-synchronization between displaying ofan image in the VR device and a user movement (which otherwise may causediscomfort of the human body) by a process shown in the dotted portionof FIG. 1.

The method, apparatus and storage medium for processing an image in a VRsystem of embodiments of the present application will be described indetail below with reference to the accompanying drawings.

A method for processing an image in a VR system of the presentapplication will be described with reference to FIG. 2.

FIG. 2 is a schematic flowchart of a method for processing an image in aVR system according to an embodiment of the present application.

As shown in FIG. 2, the method for processing an image in a VR systemaccording to this embodiment of the present application comprises thefollowing steps.

At step 201, an image to be rendered at a first timing in the VR systemis split into a first region and a second region.

The method for processing an image in a VR system provided by thisembodiment of the present application can be implemented by an apparatusfor processing an image in a VR system provided by an embodiment of thepresent application. An apparatus for processing an image in a VR systemis provided in an image processing device of the VR system to controlimage processing in the VR system. The image processing device of the VRsystem of this embodiment may be, but not limited to, VR glasses, a VRhelmet, etc.

In the present embodiment, a first timing refers to a timing at which avertical synchronization signal is generated in the VR system.Accordingly, an image to be rendered refers to an image the rendering ofwhich is started as indicated by the timing of the verticalsynchronization signal, and is also an image the displaying of which isindicated by a next vertical synchronization signal.

In the embodiment of the present application, the image to be renderedcan be split into a first region and a second region in multiplemanners. For example, the image may be split into a region of interest(the second region) and a region of non-interest (the first region)according to the gaze point of the user; alternatively, the image may besplit into a subject region (the second region) and a background region(the first region) according to a subject in the image to be rendered,which is not specifically limited in the present embodiment.

How to split a current image to be rendered in the VR system into afirst region and a second region will be described later.

At step 202, the first region of the image to be rendered is renderedaccording to first gesture data of a user at the first timing.

In this embodiment, after splitting the image to be rendered at thefirst timing in the VR system into a first region and a second region,the apparatus for processing an image in the VR system may control agesture sensor to collect gesture data at the first timing that is,first gesture data of the current user of the VR system at the firsttiming is collected and is used to render the first region of the imageto be rendered.

As an implementation, rendering the first region of the image to berendered in the present embodiment includes, for example, determining,according to the user gesture data at the first timing and gesture dataitems corresponding to various image contents in a preset image library,content to be presented in the image to be rendered, and then renderingan image corresponding to the first region based on the content to bepresented.

At step 203, the second region of the image to be rendered is renderedaccording to second gesture data at a second timing at the end of therendering of the first region.

In an example, when rendering the second region of the image to berendered, the apparatus for processing an image in the VR system maycontrol the gesture sensor to collect gesture data at a second timing atthe end of the rendering of the first region, that is, second gesturedata of the current user of the VR system at a second timing iscollected and is used to render the second region of the image to berendered, so as to obtain a rendered image.

According to the gesture data (second gesture data) of the user at thesecond timing and correspondence between preset image contents andgesture data items, this embodiment can determine content to bepresented in the image to be rendered, and then an image correspondingto the second region is rendered based on the content to be presented.

It should be noted that the refresh frequency of screens is generallybetween 60 Hz and 120 Hz, and the sampling frequency of the gesturesensor when collecting the first gesture data and the second gesturedata of the user in the present application may be 1000 Hz. It can beseen that the refresh rate of the screen is much lower than the samplingfrequency of the gesture sensor. Therefore, with the solution of thepresent application, a plurality of user gesture data items at differenttimings can be acquired in the process of rendering an image, the secondregion can be rendered by using a gesture data item as close as possibleto the display timing, as a result the rendered image can be as close aspossible to the gesture data of the user at display timing, therebyreducing the delay of the image.

Further, in this embodiment, the first region is rendered according tothe first gesture data of the user, the second region is renderedaccording to the second gesture data of the user, and the first regionand the second region may be rendered with the same resolution.Alternatively, the first region and the second region may be renderedwith different resolutions.

In an actual application, in order to improve the processing speed ofthe image to be rendered and reduce the image processing time, the firstregion and the second region of the image to be rendered may be renderedwith different resolutions.

For example, the first region is rendered with a lower resolution, andthe second region is rendered with a resolution higher than theresolution of the first region, thereby ensuring high-precisionrendering processing for the second region of the image. The renderedsecond region is more realistic and stereoscopic, and the renderingeffect of the image is improved. Through reducing the renderingresolution of the first region, the rendering processing time of thefirst region can be reduced, so that the processing speed of the imagecan be improved and the needs of the user can be satisfied.

In an embodiment of the present application, rendering the first regionof the image to be rendered comprises rendering the first region with afirst resolution; rendering the second region of the image to berendered comprises rendering the second region with a second resolution;wherein the first resolution is lower than the second resolution.

In the method for processing an image in a VR system provided in thisembodiment of the present application, an image to be rendered at afirst timing in the VR system is split into a first region and a secondregion; the first region of the image to be rendered is renderedaccording to first gesture data at the first timing; and the secondregion of the image to be rendered is rendered according to secondgesture data at a second timing at the end of the rendering of the firstregion to obtain a rendered image. Therefore, by using gesture data atdifferent timings, time-division rendering is performed on differentregions of the image to obtain a rendered image. Therefore, the timeinterval between a timing at which the rendered image is displayed and atiming at which the gesture data is collected for rendering is shortenedas much as possible, which effectively reduces the delay of the imageand improves the user experience.

As known from the above, in the embodiment of the present application,the image to be rendered can be split into a first region and a secondregion in multiple manners.

When the image is split according to a location in the image to berendered at which a subject is located, the first region is a backgroundregion, and the second region is the region at which the subject islocated. Since the region at which the subject is located is usually aregion that the user is interested in, when the region at which thesubject is located is rendered, the rendering is based on gesture dataclosest to the display timing, so that the delay of the region at whichthe subject is located in the rendered image is effectively reduced, andthe user experience is improved.

Referring to FIG. 3, a description is given of splitting an image to berendered at a first timing in the VR system into a first region and asecond region according to a location in the image to be rendered atwhich a subject is located according to an embodiment of the presentapplication.

As shown in FIG. 3, splitting an image to be rendered at a first timingin the VR system into a first region and a second region according to anembodiment of the present application comprises the following steps.

At step 301, subject recognition is performed on the image to berendered at the first timing to determine a region in the image to berendered at which a subject is located.

At step 302, a region in the image to be rendered at which the subjectis located is determined as the second region.

In an actual application, a subject in the image is usually a regionthat the user is more interested in. Therefore, the present embodimentcan perform subject recognition on the image to be rendered at a firsttiming to determine the region in the image to be rendered at which thesubject is located. The region at which the subject is located isdetermined as the second region (region of interest), and a region otherthan the region at which the subject is located is determined as thefirst region (region of non-interest).

In this embodiment, the determination of a region at which the subjectis located in the image to be rendered can be implemented by using anexisting determination method, which will not be described herein.

Further, in this embodiment, an image to be rendered at a first timingin a VR system may be split into a first region and a second regionaccording to a gaze point of the user. Referring to FIG. 4, adescription is given of splitting an image to be rendered into a firstregion and a second region based on a gaze point.

FIG. 4 is a schematic flowchart of a method for processing an image in aVR system according to another embodiment of the present application. Asshown in FIG. 4, the method for processing an image in a VR systemaccording to this embodiment of the present application comprises thefollowing steps.

At step 401, an image to be rendered at a first timing in a VR system issplit into a first region and a second region.

Referring to FIGS. 5 to 7, in the embodiment of the present application,the image to be rendered at a first timing can be split into a firstregion and a second region in multiple manners.

Referring to FIGS. 5 to 7, a description is given of splitting an imageto be rendered at a first timing in a VR system into a first region anda second region according to a gaze point of a user according to anembodiment of the present application.

As shown in FIG. 5, splitting an image to be rendered at a first timingin a VR system into a first region and a second region according to anembodiment of the present application comprises the following steps.

At step 501, when a vertical synchronization signal corresponding to theimage to be rendered at the first timing is acquired, a current gazepoint location of the user is determined.

At step 502, the gaze point location of the user at the first timing isdetermined as a reference point location of the second region.

At step 503, the second region is determined according to a presetextension mode and the reference point position.

The preset extension mode may be predetermined, or may be determinedaccording to the field of view of the user, which is not limited herein.

In an example, the present embodiment can monitor a verticalsynchronization (Vsync) signal in real time, and when the Vsync signalis detected at a first timing, acquire a gaze point location of humaneyes at the first timing, and then split an image to be rendered in theVR system into a first region and a second region according to the gazepoint location of human eyes at the first timing (corresponding to afirst gesture). To obtain a gaze point location of human eyes at a firsttiming, it may be performed by eyeball tracking to track a change of aneyeball or features around the eyeball; or it may be achieved bytracking a change in the iris angle.

That is, after acquiring a gaze point location of the user at a firsttiming, the apparatus for processing an image in a VR system maydetermine a reference point position for a second region according tothe gaze point location of the user at the first timing, and thenautomatically determine a first region and a second region according toa preset extension mode and the reference point position.

For example, as shown in FIG. 6, assume that point A is obtained as agaze point location of the user at a first timing, the apparatus forprocessing an image in a VR system may use point A as a reference pointof a second region, i.e., the point A is used as a central point, and aregion within the preset range from the gaze point A is determined as asecond region (region of interest), and the other region is determinedas a first region (region of non-interest). Referring to FIG. 6, thesplit first region (region of non-interest) is a rectangle region, andthe second region (region of interest) is a square region.

It should be noted that, in this embodiment, in addition to the shapesshown in FIG. 6, the first region (region of non-interest) and thesecond region (region of interest) obtained by splitting a current imageto be rendered may be in other shapes, such as circle, etc., which isnot limited herein.

As known from above, in the embodiment of the present application, animage to be rendered can be split into regions based on a current fieldof view of the user or optical characteristics of the VR system. Asshown in FIG. 7, splitting an image to be rendered at a first timing inthe VR system into a first region and a second region according to anembodiment of the present application comprises the following steps.

At step 701, when a vertical synchronization signal corresponding to animage to be rendered at a first timing is acquired, a location of thecurrent gaze point of a user is determined.

At step 702, a field of view of the user is determined according to adistance between the location of the current gaze point of the user anda current location of the user.

At step 703, a second region is determined according to the field ofview of the user and the current gaze point location.

In actual use, since the field of view is different when a user looks atthe same point in different locations, the content viewed by the user isdifferent. For example, for a user only looking at a table in the space,the field of view is smaller if the user is far from the table, and itwill become larger when the user approaches the table. As a result, thearea of the corresponding second region will become larger. Therefore,in the embodiment of the present application, firstly, a field of viewof the user is determined according to a current location of the userand a distance between the current location of the user and a currentgaze point of the user. Then, an image to be rendered is split accordingto the field of view and the gaze point location of the user. Thereby,the accuracy of region division is increased, and the image displayeffect can be improved.

As shown in FIG. 4, the method for processing an image in a VR system ofthis embodiment of the present application comprises the followingsteps. At step 402, a first region of an image to be rendered isrendered according to first gesture data at a first timing.

At step 403, a second region of the image to be rendered is renderedaccording to second gesture data at a second timing at the end of therendering of the first region.

According to the apparatus for processing an image in a VR system ofthis embodiment, when a second region (region of interest) of the imageto be rendered is rendered, a change from a gaze point location of theuser at a first gesture to a gaze point location of the user at a secondtiming should be in a certain range; otherwise the result of regiondivision will be different if region division is preformed on the imageto be rendered based on a gaze point location at different timing,thereby there is an error in the image obtained by rendering the secondregion (region of interest) divided based on a gaze point at the firsttiming by using second gesture data of the user collected at the secondtiming, which will reduce the user experience to a certain extent.

Therefore, in this embodiment, when the second region (region ofinterest) of an image to be rendered is rendered according to secondgesture data of the user, a gaze point location of the user at thesecond timing may be detected to ensure that when the first region(region of non-interest) and the second region (region of interest) ofthe image to be rendered are rendered, the change in the gaze pointlocation of the user is within an allowable range, so as to furtherimprove the user experience.

In some examples, a change in the gaze point location of the user may bedetermined to be within an allowable range in the following ways.

An example way may comprise: determining that a duration of rendering ofthe first region (region of interest) is less than a first threshold.The first threshold can be set based on persistence of vision.

In practical applications, in the case of observing a scene with humaneyes, it takes a short time for a light signal to be conveyed to thebrain nerves. After the end of the effect of light, a visual image doesnot disappear immediately. This residual vision is called “post-image”,and this phenomenon of vision is called “persistence of vision”. Thephenomenon of persistence of vision is the phenomenon that a visualimage produced by light on the retina remains for a short period afterthe effect of light disappears.

That is, after the first region (region of non-interest) of the image tobe rendered is rendered according to first gesture data of the user atthe first timing and when the second region (region of interest) of theimage to be rendered is rendered according to second gesture data of theuser at the second timing, if the duration of rendering the first region(region of non-interest) is less than visual persistence of the user,the gaze point of the user hardly changes when rendering the secondregion (region of interest) of the image to be rendered. In this case,the apparatus for processing an image in the VR system may render thesecond region (region of interest) of the image to be rendered accordingto the second gesture data of the user collected at the second timing.

Another exemplary way may comprise, for example, determining that adistance between a gaze point location corresponding to the secondgesture data and a gaze point location at the first timing is less thana second threshold. The second threshold may be set according to thesize of the first region (region of non-interest) split from the imageto be rendered. In this embodiment, the second threshold may be aspecific value or a range of values, which is not specifically limitedherein.

In some embodiments, after the first region (region of non-interest) ofthe image to be rendered is rendered according to first gesture data ofthe user at a first timing, if it is determined that the gaze pointlocation corresponding to second gesture data of the user at a secondtiming changes relative to gaze point location corresponding to thefirst gesture data, the gaze point location corresponding to the firstgesture data may be compared with the gaze point location correspondingto the second gesture data to obtain a distance between the gaze pointlocations corresponding to the two different timings. Then, the distanceis matched with a second threshold. If the distance is less than thesecond threshold, the change in the gaze point locations correspondingto the two different timings is considered to be in an allowable range,and the apparatus for processing an image in a VR system may render thesecond region (region of interest) of the image to be rendered accordingto the second gesture data of the user collected at the second timing.

In the method for processing an image in a VR system provided in thisembodiment of the present application, an image to be rendered at afirst timing in the VR system is split into a first region and a secondregion; the first region of the image to be rendered is renderedaccording to first gesture data at the first timing; and the secondregion of the image to be rendered is rendered according to secondgesture data at a second timing at the end of the rendering of the firstregion to obtain a rendered image. Therefore, by using gesture data atdifferent timings, time-division rendering is performed on differentregions of the image to obtain a rendered image. Therefore, the timeinterval between a timing at which the rendered image is displayed and atiming at which the gesture data is collected for rendering is shortenedas much as possible, which effectively reduces the delay of the image.Since the second region which is a region of interest or a subjectregion is closer in time to the display timing, and the first region isa region of non-interest or a background region, the delay has littleeffect on the user, thereby improving the user experience.

As known from above, in the embodiment of the present application, animage to be rendered is split into a first region and a second region;the first region is rendered according to first gesture data of the userat a first timing; and the second region is rendered according to secondgesture data of the user at a second timing to obtain a rendered image.

In actual use, in order to further reduce the delay of the image, in anembodiment of the present application, a TimeWarp process may be furtherperformed on an image corresponding to the first region as rendered andan image corresponding to the second region as rendered after renderingthe first region of the image to be rendered. A method for processing animage in a VR system according to an embodiment of the presentapplication will be described with reference to FIG. 8.

FIG. 8 is a schematic flowchart of a method for processing an image in aVR system according to a further embodiment of the present application.

As shown in FIG. 8, the method for processing an image in a VR systemaccording to this embodiment of the present application comprises thefollowing steps.

At step 801, an image to be rendered at a first timing in the VR systemis split into a first region and a second region.

At step 802, the first region of the image to be rendered is renderedaccording to first gesture data at the first timing.

At step 803, the second region of the image to be rendered is renderedaccording to second gesture data at a second timing at the end of therendering of the first region.

For the implementation process and the principle of steps 801-803,reference can be made to the detailed description of the foregoingembodiments, which will not be described herein.

At step 804, a TimeWarp (TW) process is performed on an imagecorresponding to the first region as rendered and an image correspondingto the second region as rendered.

The TW process is a technique for image frame correction. In the case ofusing a VR device, there is a delay in scene rendering due to the fastmovement of the user's head, that is, it has no time to render an imagecorresponding to a certain location when the user turns his/her head tothe location, or only a previous image frame can be rendered. TWprocessing is used to solve the delay problem by wrapping an image framebefore sending it to the display.

The TW processing in this solution may be a direction-based TW process,which only corrects a change in the rotational gesture of the head.Thus, it does not need too much system resources to combine a deformedimage through the above processing. For a complex scene, a new image canalso be obtained with less calculation.

A method for processing an image in a VR system according to anembodiment of the present application will be described with referenceto FIGS. 9 and 10.

FIG. 9 is a process of rendering and displaying an image to be renderedin the related art. Referring to FIG. 9, when the apparatus forprocessing an image in a VR system detects the vertical synchronization(Vsync) signal of the N-1th frame, that is, the first Vsync signal inthe figure, it can render the N-1 frame while controlling to start thedisplaying of the N-2th frame (not shown). When rendering the N-1thframe, first, gesture data at timing t00 (that is, the timing of thefirst Vsync signal) is acquired, and the N-1th frame is rendered withthis gesture data. At the end of the rendering of the N-1th frame,TimeWarp processing (i.e., TW processing) is performed on the N-1thframe. When a second Vsync signal is detected, the apparatus controls tostart the displaying of the N-1th frame until the displaying iscompleted (for example, t₁₁). In addition, when the second Vsync signalis detected, rendering of the Nth frame is started, and so on. Accordingto FIG. 9, in the entire process from rendering the N-1 frame todisplaying its image above, the delay in the VR system is: Δt=t₁₁−t₀₀,that is, the image displayed on the system is an image that is renderedwith gesture data at timing t₀₀ which is Δt=t₁₁−t₀₀ before timing t₁₁.However, the human body has changed from the first gesture at timing t₀₀to the third gesture at timing t₁₁. Due to the system delay, humandiscomfort is caused by viewing, at the third gesture timing, an imagethat should be displayed at the first gesture timing.

It should be noted that the time interval from the start of displaying aframe to the completion of display of the frame in the figure isdetermined by the screen refresh frequency, which is generally aninterval for data writing on the display screen and is less than theactual refresh time of one frame. That is, the interval from t₁₀ to t₁₁is less than the time interval between two vertical synchronizationsignals.

In order to reduce the delay of the VR system from the rendering processto the completion of the screen display, the process of rendering anddisplaying an image to be rendered in the embodiment of the presentapplication is shown in FIG. 10. Referring to FIG. 10, when theapparatus for processing an image in a VR system detects the verticalsynchronization (Vsync) signal of the N-1th frame, that is, the firstVsync signal in the figure, it can render the N-1 frame whilecontrolling to start the displaying of the N-2th frame (not shown). Whenrendering the N-1th frame, first, an eye gaze point location at thefirst timing (corresponding to a first gesture) is obtained by eyeballtracking, and the N-1th frame is automatically split into a first region(region of non-interest, RONI) and a second region (region of interest,ROI) according to the gaze point location. Then, gesture data (firstgesture) at timing t₀₀ (the timing at which the first Vsync signal isobtained) is obtained, which is then used to render a region ofnon-interest of the N-1th frame. At the end of the rendering of theregion of non-interest of the N-1th frame, gesture data (second gesture)at timing t₀₁ (a timing at which the rendering of the region ofnon-interest of the N-1th frame ends), which is then used to render aregion of interest of the N-1th frame. When the rendering of the N-1thframe is completed, TW processing is performed on an image correspondingto the region of non-interest as rendered and an image corresponding tothe region of interest as rendered. Then, at the timing of the secondVsync signal, the apparatus controls to start the display of the N-1thframe until the completion of the display (for example, t₁₁). Inaddition, at the timing of the second Vsync signal, the apparatus startsthe rendering of the Nth frame, and so on. According to FIG. 10, in theentire process from rendering the N-1 frame to displaying its image ofthis embodiment, the delay in the VR system is: Δt′=t₁₁−t₀₁, that is,the image displayed on the system is an image that is rendered withgesture data at timing t₀₀ which is Δt′=t₁₁−t₀₁ before t₁₁.

Through comparing the delay in the entire process from rendering animage to be rendered to displaying the rendered image in the VR systemof the embodiment of the present application with the delay in theentire process from rendering an image to be rendered to displaying therendered image in the VR system of the related art, it can be seen thatthe embodiment of the present application can reduce the delay in theentire process from rendering an image to be rendered to displaying therendered image in the VR system by an amount of:T=Δt′−Δt=(t₁₁−t₀₁)−(t₁₁−t₀₀)=t₀₀−t₀₁.

Thus, through performing an image to be rendered with the method forprocessing an image in a VR system of the embodiment of the presentapplication, compared with the method for processing an image to berendered in the related art, the delay of image display can be furtherreduced, so that the displayed image is matched with the movement of theuser better, the demand of the user can be satisfied, and the userexperience can be improved. Moreover, in the embodiment of the presentapplication, the image to be rendered is split into a region of interestand a region of non-interest to perform time-division rendering on theregion of interest and the region of non-interest with gesture data atdifferent timings in the rendering of the region of interest and theregion of non-interest. Therefore, the time interval between a timing atwhich the rendered image is displayed and a timing at which the gesturedata is collected for rendering is shortened as much as possible, whicheffectively reduces the delay of the image. In addition, since theregion of interest is closer in time to the display timing, and theregion of non-interest does not have a significant impact on the viewingof the user, the user experience can be improved.

An apparatus for processing an image in a VR system is further providedin the present application. FIG. 11 is a schematic structural diagram ofan apparatus for processing an image in a virtual reality systemaccording to an embodiment of the present application.

As shown in FIG. 11, the apparatus for processing an image in a VRsystem of this embodiment of the present application comprises: acentral processing unit (CPU) 901 and a graphics processing unit (GPU)902.

The central processing unit is configured to split an image to berendered at a first timing in the virtual reality system into a firstregion and a second region.

The graphics processing unit 902 is configured to, according to firstgesture data of a user at a first timing, render the first region of theimage to be rendered.

The graphics processing unit 902 is further configured to, according tosecond gesture data of the user at a second timing at the end of therendering of the first region, render the second region of the image tobe rendered.

The gesture data of the user can be obtained by a somatosensoryrecognition component in the VR system.

The somatosensory recognition component comprises at least one of: agyroscope, a gravity sensor, a magnetic sensor, an infrared sensor, etc.

As an implementation of the present application, the first timing is atiming when a vertical synchronization signal is generated in the VRsystem.

As an implementation of the present application, the GPU 902 is furtherconfigured to: perform a TimeWarp process on an image corresponding tothe first region as rendered and an image corresponding to the secondregion as rendered.

As an implementation of the present application, the GPU 902 is furtherconfigured to: render the first region with a first resolution; renderthe second region with a second resolution; wherein, the firstresolution is lower than the second resolution.

As an implementation of the present application, the CPU 901 isconfigured to: perform subject recognition on the image to be renderedat the first timing to determine a region in the image to be rendered atwhich a subject is located; determine the region in the image to berendered at which the subject is located as the second region.

As an implementation of the present application, the CPU 901 isconfigured to: determine a gaze point location of the user at the firsttiming as a reference point position of the second region; determine thesecond region according to a preset extension mode and the referencepoint position.

As an implementation of the present application, the CPU 901 isconfigured to: determine a field of view of the user according to adistance between a current gaze point location of the user and a currentlocation of the user; determine the second region according to the fieldof view of the user and the current gaze point location.

As an optional implementation of the present application, the CPU 901 isfurther configured to: determine that a duration of rendering of thefirst region is less than a first threshold; or determine that adistance between a gaze point location corresponding to the secondgesture data and a gaze point location at the first timing is less thana second threshold.

As shown in FIG. 12, in another embodiment of the present application,the apparatus 200 for processing an image in a VR system furthercomprises: memory 210 and a processor 220. Wherein, a computer programis stored in the memory 210; when the computer program stored in thememory 210 is invoked and executed by the processor 220, it implementsthe method for processing an image in a VR system according to the firstembodiment of the present application.

In an optional implementation, as shown in FIG. 13, the apparatus 200for processing an image in a VR system may further include: memory 210and a processor 220, and a bus 230 connecting different components(including the memory 210 and the processor 220), a computer programstored on the memory 210, which when executed by the processor 220implements method for processing an image in a VR system according tothe first embodiment of the present application.

The bus 230 represents one or more of several types of bus structures,including a memory bus or memory controller, a peripheral bus, agraphics acceleration port, a processor, or a local bus using any of avariety of bus structures. For example, these architectures include, butare not limited to, an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MAC) bus, an Enhanced ISA Bus, a Video ElectronicsStandards Association (VESA) local bus, and peripheral componentinterconnects (PCI) bus.

The computer device 200 typically includes a variety of computer devicereadable media. These media can be any available media that can beaccessed by computer device 200, including both volatile and nonvolatilemedia, removable and non-removable media.

The memory 210 may also include computer system readable media in theform of volatile memory, such as random access memory (RAM) 240 and/orcache memory 250. The computer device 200 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example, a storage system 260 can be used to read fromand write to non-removable, non-volatile magnetic media (not shown inFIG. 13, generally referred to as “hard disk drives”). Although notshown in FIG. 13, a magnetic disk driver for reading from and writing toa removable non-volatile disk (such as a “floppy disk”), and an opticaldisk driver for reading from and writing to a removable non-volatileoptical disk (such as a CD-ROM, DVD-ROM, or other optical media) may beprovided. In these cases, each driver can be coupled to the bus 230 viaone or more data medium interfaces. The memory 210 can include at leastone program product having a set (e.g., at least one) of program modulesconfigured to perform the functions of the various embodiments of thepresent application.

A program/utility 280, having a set (at least one) of program modules270, may be stored in the memory 210 by way of example, and notlimitation, as well as an operating system, one or more applicationprograms, other program modules, and program data. Each of the operatingsystem, one or more application programs, other program modules, andprogram data or some combination thereof, may include an implementationof a networking environment. The program modules 270 generally carry outthe functions and/or methodologies of embodiments of the presentapplication as described herein.

The computer device 200 may also communicate with one or more externaldevices 290 such as a keyboard, a pointing device, a display 291, etc.;one or more devices that enable a consumer to interact with the computerdevice 200; and/or any devices (e.g., network card, modem, etc.) thatenable the computer device 200 to communicate with one or more othercomputing devices. Such communication can occur via I/O interfaces 292.Still yet, the computer device 12 can communicate with one or morenetworks such as a local area network (LAN), a general wide area network(WAN), and/or a public network (e.g., the Internet) via a networkadapter 293. As depicted, the network adapter 293 communicates with theother components of the computer device 200 via the bus 230. It shouldbe understood that although not shown, other hardware and/or softwarecomponents could be used in conjunction with the computer device 200.Examples include, but are not limited to: microcode, device drivers,redundant processing units, external disk drive arrays, RAID systems,tape drives, and data archival storage systems, etc.

It should be noted that the foregoing explanation of the embodiment ofthe method for processing an image in a VR system is also applicable tothe apparatus for processing an image in a VR system of this embodiment,the implementation principle thereof is similar, and will not bedescribed in detail herein.

In the apparatus for processing an image in a VR system provided in thisembodiment of the present application, an image to be rendered at afirst timing in the VR system is split into a first region and a secondregion; the first region of the image to be rendered is renderedaccording to first gesture data at the first timing; and the secondregion of the image to be rendered is rendered according to secondgesture data at a second timing at the end of the rendering of the firstregion to obtain a rendered image. Therefore, by using gesture data atdifferent timings, time-division rendering is performed on differentregions of the image to obtain a rendered image. Therefore, the timeinterval between a timing at which the rendered image is displayed and atiming at which the gesture data is collected for rendering is shortenedas much as possible, which effectively reduces the delay of the imageand improves the user experience.

The present application further provides a computer-readable medium. Thecomputer-readable storage medium has a computer program stored thereon,which when executed by a processor, implements the method for processingan image in a VR system according to the first embodiment of the presentapplication.

In an optional implementation, this embodiment can employ anycombination of one or more computer readable media. The computerreadable medium can be a computer readable signal medium or a computerreadable storage medium. The computer readable storage medium can be,for example, but not limited to, an electronic, magnetic, optical,electromagnetic, infrared, or semiconductor system, apparatus, ordevice, or any combination of the above. More specific examples(non-exhaustive lists) of computer readable storage medium include:electrical connection with one or more wires, portable computer disk,hard disk, random access memory (RAM), read only memory (ROM), erasableprogrammable read only memory (EPROM or flash), fiber optics, portablecompact disk Read only memory (CD-ROM), optical storage device, magneticstorage device, or any suitable combination of the foregoing. In thisdocument, a computer readable storage medium can be any tangible mediumthat can contain or store a program, which can be used by or inconnection with an instruction execution system, apparatus or device.

A computer readable signal medium may include a data signal that ispropagated in the baseband or as part of a carrier, carrying computerreadable program code. Such propagated data signals can take a varietyof forms including, but not limited to, electromagnetic signals, opticalsignals, or any suitable combination of the foregoing. The computerreadable signal medium can also be any computer readable medium otherthan a computer readable storage medium, which can transmit, propagate,or transport a program for use by or in connection with the instructionexecution system, apparatus, or device.

Program code embodied on a computer readable medium can be transmittedby any suitable medium, including but not limited to wireless, wire,fiber optic cable, RF, etc., or any suitable combination of theforegoing.

The computer program code for executing operations of the presentapplication may be complied by any combination of one or more programdesign languages, the program design languages including object-orientedprogram design languages, such as Java, Smalltalk, C++, etc, as well asconventional procedural program design languages, such as “C” programdesign language or similar program design language. A program code maybe completely or partly executed on a user computer, or executed as anindependent software package, partly executed on the user computer andpartly executed on a remote computer, or completely executed on a remotecomputer or server. In the latter circumstance, the remote computer maybe connected to the user computer through various kinds of networks,including local area network (LAN) or wide area network (WAN), orconnected to external computer (for example using an internet serviceprovider via Internet).

The present application further provides a computer program. Whenexecuted by a processor, the computer program implements the method forprocessing an image in a VR system according to the first embodiment ofthe present application.

In the present application, the terms “set”, “connection” and the likeshould be interpreted broadly, for example, a connection may be amechanical connection or electrical connection, unless otherwiseexplicitly stated and defined; it may be direct connection or indirectconnection through an intermediate medium, and may be internalcommunication of two elements or an interaction relationship of twoelements, unless explicitly defined otherwise. For those of ordinaryskill in the art, the specific meanings of the above terms in thisapplication can be interpreted according to particular situations.

In the description of this specification, reference throughout thisspecification to “one embodiment”, “some embodiments”, “illustrativeembodiments”, “examples”, “specific examples”, or “some examples” meansthat a particular feature, structure, material or characteristicdescribed in connection with the embodiment is included in at least oneembodiment of the present application.

Furthermore, the terms “first” and “second” are used for descriptivepurposes only, and are not to be construed as indicating or implyingrelative importance or implicitly indicating the number of indicatedtechnical features. Thereby, features defined by “first”, “second” mayexpressly or implicitly include one or more of the features.

Any process or method description in the flowcharts or otherwisedescribed herein may be interpreted as representing a module, segment orportion of code that includes one or more executable instructions forimplementing the steps of a particular logical function or process. Andthe scope of the preferred embodiments of the present applicationincludes additional implementations, in which the functions may not beperformed in the order shown or discussed, including in a substantiallysimultaneous manner or in an opposite order depending on the functionsinvolved, as understood by those skilled in the art to which theembodiments of the present application belong.

It should be understood that portions of the present application can beimplemented in hardware, software, firmware, or a combination thereof.In the above-described embodiments, multiple steps or methods may beimplemented in software or firmware stored in a memory and executed by asuitable instruction execution system. For example, if implemented inhardware, as in another embodiment, it can be implemented with any oneor a combination of the following technologies, which are each wellknown in the art: a discrete logic circuit(s) having logic gates forimplementing logic functions upon data signals, an application specificintegrated circuit having appropriate combinational logic gates, aprogrammable gate array(s) (PGA), a field programmable gate array(FPGA), etc.

The common technical personnel in the field can understand that all orsome steps in the embodiments can be completed by the means thatrelevant hardware is instructed by a program, the program can be storedin a computer readable storage medium, and the program comprises any oneor combination of the steps in the embodiments when being executed.

The above mentioned storage medium may be a read only memory, a magneticdisk or an optical disk, etc. Although embodiments of the presentdisclosure have been illustrated and described above, it will beunderstood that the above-described embodiments are exemplary and not tobe construed as limiting the present application. Those of ordinaryskill in the art may make variations, modifications, substitutions, andvariations to the above described embodiments within the scope of thepresent application.

What is claimed is:
 1. A method for processing an image in a virtualreality system, comprising: splitting an image to be rendered at a firsttiming in the virtual reality system into a first region and a secondregion by determining a gaze point location of the user at the firsttiming as a reference point position of the second region, determiningthe second region according to a preset extension mode and the referencepoint position, determining a field of view of the user according to adistance between a current gaze point location of the user and a currentlocation of the user, and determining the second region according to thefield of view of the user and the current gaze point location; renderingthe first region of the image to be rendered according to first gesturedata of a user at the first timing; rendering the second region of theimage to be rendered according to second gesture data at a second timingat an end of the rendering of the first region; and before rendering thesecond region of the image to be rendered, determining that a distancebetween a gaze point location corresponding to the second gesture dataand the gaze point location at the first timing is less than athreshold.
 2. The method according to claim 1, wherein the first timingis a timing at which a vertical sync signal is generated in the virtualreality system.
 3. The method according to claim 1, further comprising:after rendering the second region of the image to be rendered,performing a TimeWarp process on an image corresponding to the firstregion as rendered and an image corresponding to the second region asrendered.
 4. The method according to claim 3, wherein splitting theimage to be rendered at the first timing in the virtual reality systeminto the first region and the second region comprises: performingsubject recognition on the image to be rendered at the first timing todetermine a region in the image to be rendered at which a subject islocated; and determining the region in the image to be rendered at whichthe subject is located as the second region.
 5. The method according toclaim 3, wherein splitting the image to be rendered at the first timingin the virtual reality system into the first region and the secondregion comprises: determining a gaze point location of the user at thefirst timing as a reference point position of the second region; anddetermining the second region according to a preset extension mode andthe reference point position.
 6. The method according to claim 1,wherein: rendering the first region of the image to be renderedcomprises rendering the first region with a first resolution; renderingthe second region of the image to be rendered comprises rendering thesecond region with a second resolution; and the first resolution islower than the second resolution.
 7. The method according to claim 1,wherein splitting an image to be rendered at a first timing in thevirtual reality system into a first region and a second regioncomprises: performing subject recognition on the image to be rendered atthe first timing to determine a region in the image to be rendered atwhich a subject is located; and determining the region in the image tobe rendered at which the subject is located as the second region.
 8. Anapparatus for processing an image in a virtual reality system,comprising: a central processing unit configured to split an image to berendered at a first timing in the virtual reality system into a firstregion and a second region by determining a gaze point location of theuser at the first timing as a reference point position of the secondregion, determining the second region according to a preset extensionmode and the reference point position, determining a field of view ofthe user according to a distance between a current gaze point locationof the user and a current location of the user, and determining thesecond region according to the field of view of the user and the currentgaze point location; and a graphics processing unit configured to:render the first region of the image to be rendered according to firstgesture data of a user at the first timing; render the second region ofthe image to be rendered according to second gesture data at a secondtiming at an end of the rendering of the first region; and beforerendering the second region of the image to be rendered, determine thata distance between a gaze point location corresponding to the secondgesture data and the gaze point location at the first timing is lessthan a threshold.
 9. The apparatus according to claim 8, wherein thegraphics processing unit is further configured to: perform a TimeWarpprocess on an image corresponding to the first region as rendered and animage corresponding to the second region as rendered after rendering thesecond region of the image to be rendered.
 10. The apparatus accordingto claim 8, wherein: in order to render the first region of the image tobe rendered, the graphics processing unit is configured to render thefirst region with a first resolution; in order to render the secondregion of the image to be rendered, the graphics processing unit isfurther configured to render the second region with a second resolution;and the first resolution is lower than the second resolution.
 11. Theapparatus according to claim 8, wherein in order to split the image tobe rendered at the first timing in the virtual reality system into thefirst region and the second region, the central processing unit isfurther configured to: perform subject recognition on the image to berendered at the first timing to determine a region in the image to berendered at which a subject is located; and determine the region in theimage to be rendered at which the subject is located as the secondregion.
 12. A non-transitory computer-readable storage medium on which acomputer program is stored, which when executed by a processor causedthe processor to: split an image to be rendered at a first timing in thevirtual reality system into a first region and a second region bydetermining a gaze point location of the user at the first timing as areference point position of the second region, determining the secondregion according to a preset extension mode and the reference pointposition, determining a field of view of the user according to adistance between a current gaze point location of the user and a currentlocation of the user, and determining the second region according to thefield of view of the user and the current gaze point location; renderthe first region of the image to be rendered according to first gesturedata of a user at the first timing; render the second region of theimage to be rendered according to second gesture data at a second timingat an end of the rendering of the first region; and before rendering thesecond region of the image to be rendered, determine that a distancebetween a gaze point location corresponding to the second gesture dataand the gaze point location at the first timing is less than athreshold.